1. Field of the Invention
The present invention relates to the protection of computer systems. More particularly, the present invention relates to hooking of runtime processes.
2. Description of the Related Art
The necessity of hooking runtime processes arises in various scenarios and situations like debugging, troubleshotting, profiling, extending functionality, etc. The challenge is to be able to successfully hook the function without interrupting the flow of execution.
When hooking a function via a prologue overwrite, the case that another thread was executing the prologue of the function that was hooked at the time it was hooked should be considered. With modern processors, the cache will become invalidated once the prologue is overwritten so that the CPU will execute the modified instructions. However, if it had executed only part of the prologue, this may create an invalid state that will crash the thread or the process.